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ABSTRACT 


This thesis documents the collection and nical of experimental data 
used to compare and contrast three methods of evaluating attribute 
choices that have no natural measurement basis. Attribute choice 
suniisetion Is prerequisite to ranking and evaluating multi-attripbute decision 
alternative sets. Data for the methods center on subjective inputs from 
subject matter experts. The particular experiment collected response data 
from 27 experts in the United States Navy LAMPS (Light Airborne Multipurpose 
system) helicopter community attached to the Naval Postgraduate School. 
The pilots compared three helicopter systems (attribute choices) in each of 
four system categories (attributes); weapons, navigation systems, 
communication systems, and sensors. The complete procedure would 
evaluate every feasible helicopter system suite (decision alternative set), 
each set composed of one attribute choice from every attribute, facilitating 
ordinal ranking of the sets. Thesis results present consistency analyses of the 
experts’ responses within, and between, the three methods of divine 


attribute choice values. 
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EXECUTIVE SUMMARY 


| Military decision makers are constantly challenged to make optimal 
choices between competing alternatives. This challenge is made even 
more demanding in that, often, competing alternatives are very pseeielins 
and have ie natural measurement basis from which to conduct direct 
comparisons. By sufficiently dividing each of the complex alternatives into 
characteristics, called attributes, common to all, or at least most, of the 
alternatives, direct comparisons and evaluations of each alternative's 
particular attribute, called attribute choices, might be made. By 
weighting the importance of each of the atiriputes, a value for each of 
the competing alternatives can be tivities and direct quantitative 
comparisons made. 

This thesis focuses on the first major step of the overall process; 
evaluating attribute choices with no natural measurement basis. Three 
methods of determining attribute choice values which use subject matter 
expert subjective inputs were compared and contrasted. Experiment 
response data from 27 experts in the United States Navy LAMPS (Light 
Airborne Multipurpose System) helicopter community were examined for 


response consistency within and between the three methods. 


xV 


The three methods examined to evaluate attribute choices 
involved direct evaluation via a linear scale, and indirect evaluation. via 
pairwise comparisons and indifference probabilities. Derived data values 
were plotted versus directly obtained values. The amount and 
characteristic behavior (bias) of the derived values’ dispersion from the 
directly obtained values within and between each of the methods were 
calculated and compared. 

The smaller the magnitudes of dispersion and bias, the more 
consistent the responses. The intra-method consistency checks in the 
table indicate that while the pairwise comparison method showed the 
least dispersion, it was more susceptible to bias than the indifference 


probability method. The same conclusions seemingly can be drawn from 
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fference Probabilities 

Inter-method Consistency Checks Dispersion, d_,,| Bias, b,.,,, 

Marking a Linear Scale vs | 0.0361 -0.2264 
Pairwise Comparisons | 

Marking a Linear Scale vs 0.2968 -0.0862 
Indifference Probabilities | 

Pairwise Comparisons vs | 0.3962 -0.2052 
Indifference Probabilities 
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nira-method Consistency Checks | Dispersion,d..| Bias,b., 
















the inter-method consistency checks; however, method interaction 
prevents definite inferences from being made. 

Additional consistency checks seemed to imply the experts were 
risk prone when they perceived there was little to loose and much to 
gain. Conversely, the experts were risk averse when they perceived there 


was much to loose and little to gain. 
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i. INTRODUCTION 


A. THESIS SCOPE AND GOALS 

This thesis documents the collection and analysis of experimental 
data used to compare and contrast three methods of evaluating 
attribute choices that have no natural measurement basis. Attribute 
choice evaluation is prerequisite to ranking and evaluating multi-attribute 
decision alternative sets. Data for the methods center on subjective 
inputs from subject matter experts. The particular experiment collected 
response data from 27 experts in the United States Navy LAMPS (Light 
Airborne Multipurpose System) helicopter community attached to the 
Naval Postgraduate School. 

These LAMPS pilots were asked to compare three helicopter systems 
(attripnute choices) in each of four system categories (attributes); weapons, 
navigation systems, communication systems, and sensors. The complete 
procedure would evaluate every feasible helicopter system suite 
(decision alternative sets), facilitating their ordinal ranking. It should be 
emphasized that this thesis concentrates on consistency analyses within 
and between the three methods of determining attribute choice values. 
It is not the goal nor intent of this thesis to evaluate LAMPS helicopter 


system suites. 


B. THESIS OUTLINE 

Chapter || provides definitions of multi-attrioute decision alternative 
sets, attributes, and atiribute choices, and an overview of how these 
elements are combined to evaluate decision alternative sets. The 


employment of subjective inputs from subject matter experts is then 


! 


discussed. The bulk of the chapter concentrates on describing the three 
attribute choice evaluation methods. Techniques used to apply the 
methods are discussed, as are ways to conduct intra and inter-method 
consistency checks. 

Chapter Ill highlights the experimental approach used to collect 
the thesis data. Also included is a brief discussion of the history and 
current mission applications of LAMPS. This discussion is meant to give the 
reader unfamiliar with LAMPS some insight into the experimental survey 
setup and question design. In view of the broad range and variety of 
missions performed by LAMPS, the importance of and reasoning behind 
presenting the pilots with a standardized, albeit generalized, mission are. 
also discussed. The specific attributes and attribute choices used for the 
experiment are then presented, along with samples of the 81 possible 
helicopter system suites that could be created. The chapter finishes with 
an overview of the survey response form and guide, and a discussion on 
the actual administration of the surveys. 

The methods used to analyze the data, specifically intra and 
inter-method consistency checks, are described in Chapter IV. 
Consistency check plots, along with the two parameters used to 
quantitatively measure consistency, are derived and results for five main 
data subsets are presented. Tables summarizing the consistency check 
parameters for 28 various data subsets conclude the chapter. 

Chapter V concentrates on interpreting the consistency check 
parameters developed in Chapter IV for the five main data subsets. 
Some consistency checks that yielded unexpected results are analyzed in 


greater detail and utility function theory applied in an attempi to explain 








the results. The thesis closes with discussions on method applicability, 


experiment conclusions and suggested areas for future study. 
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Il. MULTI-ATTRIBUTE DECISION ALTERNATIVE SETS WITH NO NATURAL 
MEASUREMENT UNITS 


Chapter Il opens by providing definitions of multi-attripute decision 
alternative sets, attributes, and attribute choices, and an overview of how 
these elements are combined to evaluate decision alternative sets. The 
employment of subjective inputs from subject matter experts is then 
discussed. The bulk of the chapter concentrates on describing the three 
attribute choice evaluation methods. Techniques used to apply the 
methods are discussed, as are ways to conduct intra and inter-method 


consistency checks. 


A. DEFINITIONS 
To fully comprehend the evaluation methods described later in this 


chapter, the following definitions should be thoroughly understood: 


¢Multi-Attribute Decision Alternative Set: Also called a 
decision alternative, alternative set, or set, a multi-attripute 
decision alternative set is a complex object upon which a 
decision maker wants to place a quaniitative value in order 
to directly compare if with similar complex objects. For 
example, an auto dealership may present a perspective car 
buyer {i.e., a decision maker) with dozens of alternative sets 
(i.e., cars) that he wants to evaluate. 

eAttribute: An oftiribufe is an alternative set characteristic 
common to all (or at least most) of the sets considered for 


evaluation and ranking. For example, attribute candidates 
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for new cars could include gas mileage and exterior color. 
Note that some attributes (e.g., gas mileage) are much 
easier to quantify than others (exterior color). 
eAttribute Choice: An atiribute choice, or choice, is the 
particular item or characteristic of a certain attribute in each 
alternative set. For the car example, attribute choices for 
the exterior color attribute may be green, red, and black for 

some car models, and silver, dark blue, red, and turquoise for 
others. The gas mileage atiribute choice for one model of 
car might be 17.9 miles per gallon (mpg) and 24.1 mpg fora 
different model. Note that the number of attribute choices 
need not be the same for each attribute (six exterior color 
attribute choices, but only two gas mileage choices). Also 
note that not all alternative sets may be available or feasible 


(the dealer may not have a red car that gets 24.1 mpg). 


B. PROBLEM STATEMENT AND FORMULATION 

This thesis examines and checks the consistency of three attribute 
choice evaluation methods that are prerequisite for evaluating 
alternative sets that have no natural basis for comparison. The absence 
of a natural comparison basis creates questions of how to place an 
overall value on alternative sets and, ultimately, how to consistently 
choose the best alternative set. | 

Rather than try to compare alternative sets considering all attributes 
simultaneously, the sets are broken down into single attribute choices 


upon which the methods are then used to derive attribute choice values 








and attribute weighting factors. From these, an overall value for each 


alternative set is then calculated. 

Assume that there are m alternative sets and that each alternative 
set has n attributes. The alternative sets are indexed by j = 1, 2,...,m, and 
the attributes by i = 1, 2, ...,n. Let V,(j) be the amount of value obtained 
from attribute i in alternative set j. The overall value of set j is assumed to 


be given by Equation (1): 


V(j) = Diet a, V. (j) foranyj=1, 2, ..,m, where (1) 
V(j) = overall value for alternative set j: 


a, = weighting factor for attribute i; a, is the amount of set j's 
overall value obtained from a unit of attribute i: 

a, >Qif, for attribute i, bigger is better [Ref. 1, Table 7.2]; and, 

a, <O/if, for attribute i, smaller is better; and, 


V (j) = value of attribute i for alternative set j. 


Given the V,(j)'s and a, 's, the overall numerical values for each of 
the alternative sets, the V(j)'s, can be calculated. Once the V(j)'s have 
been found, they can be ranked to find the alternative set with the 


greatest overall value. Mathematically the objective is to find 


Max, { V(j) = a, V,()) + a, Vo{j) +... +a, Vif) }- | (2) 


Appendix A provides a more in-depth discussion of the nomenclature and 


indexing relationships used throughout this thesis. If the reader continues to have 
questions regarding the interrelationships of Equation (1) and (2) components, he 
should refer to Appendix A at this time. 


/ 


Numerical values for each of the V,(j)'s and a, 's, however, are 
generally not readily available. Before the alternative sets can be 
ranked, these values must be determined. 

This thesis fakes an experimental approach to test decision maker 
response consistency within and between three methods for deriving the 
V.(j)'s. For future study, data was also collected to examine expert 


response consistency in deriving a, 's. 


C. SUBJECT MATTER EXPERT SUBJECTIVE INPUTS 

Deriving a value for an attribute choice may be a matter of faking 
a simple physical measurement, such as temperature or volume. Many 
attributes, however, do not lend themselves as readily fo evaluation, or 
are very time consuming or expensive to evaluate. Attribute weighting 
factors are even less easily obtained. 

This thesis compares several methods for deriving attribute choice 
values. The methods make use of subjective inputs, from subject matter 
experts (also called decision makers, or experts). For each attribute, 
experts are asked to rank the attribute choices from the one with the most 
preferred level to the one with the least preferred level. 

Having ordinally ranked the attribute choices for each attribute, 


experts are asked to compare the choices in the following three ways: 


For each attribute: 
® assign a value to each attribute choice from a linear scale 
(Method 1); 


e make pairwise comparisons of attribute choices by 








assigning a number to the ratio of their values (Method 2): 
and, 

¢ find an indifference probability for a gamble between the 
best and worst attribute choices and a certainty of 


obtaining an intermediate attribute choice (Method 3). 


Using Method | the V,(j)'s are obtained directly. Methods 2 and 3 
were designed to build in consistency checks within each method. 

Experimental results are presented in Chapter IV that demonstrate 
the levels of consistency within Methods 2 and 3, followed by 


inter-method consistency checks between Methods 1 and 2, 1 and 3, and 
2 and 3. 


D. METHOD STRUCTURES AND CONSISTENCY CHECKS 

For each attribute, an expert is asked fo provide a series of 
subjective inputs as to which choice possesses the most preferred level of 
that attribute, the second most preferred level, and so on down to the 
attribute choice with the least preferred level of that attribute. The expert 
then uses the three methods to quantify the degree to which each 
choice's attribute level differs from the remaining choices. Finally, these 
responses, and those to develop attribute weighting factors, could be 
applied to evaluate decision alternatives. Again, this thesis does not 
attempt to evaluate decision alternatives, rather, the degree of expen 
response consistency within and between the three attribute choice 


evaluation methods is examined. 





1. Data Acquisition Steps; Comparison Methods and Consistency 
Checks 
A series of responses is required of the expert in order to collect 
data to derive attribute choice values. | 

a. Step I: Attribute Choice Subjective Rankings 

For each attribute, the expert ranks the attribute choices from 
the one with the most preferred level (i.e., the best choice) of an 
attribute, to the one with the least preferred level {i.e., the worst choice) 
of that attribute. For example: if the attribute under consideration is gift 
stature, and the attribute choices are a $5.00 gift, $20.00 gift, or $100.00 
gift, then the expert would likely rank the $100.00 gift attribute choice as 
having the most preferred level of gift stature attribute, the $20.00 gift as 
having less of the attribute than the $100.00 gift, but more than the $5.00 
gift, and the $5.00 gift last for having the least preferred level of the 
attribute gift stature. Symbolically, $100.00 gift > $20.00 gift - $5.00 gift, 
where the symbol > indicates the choice to the left of the symbol is 
preferred to the choice to the right of the symbol. The symbol does not 
represent greater than. 

Responses for Step 1 are inherently consistent. They would 
become inconsistent only if the expert later decided his initial rank order 
did not correctly represent his opinion. If the expert changes his initial 
rank order before he completes all the data collection steps, the data 
collection process must be restarted at Step |. 

b. Step 2: Marking a Linear Scale (Method 1) 

The expert is next asked to quantify the attribute choices 
using the three methods discussed earlier in Section C of this chapter. 
Using the first method, designated Method 1, the expert derives attribute 

10 








choice values directly by marking a linear scale labeled from zero, 
representing an attribute choice with an absolutely worthless value, to 
100, representing an attribute choice with the best possible value. 

Figure ] shows an example of such a scale and a possible set of responses 


to the gift stature attribute example introduced in the previous section. 


Absolutely worthless . Best ee wake 
level of gift stature Expert S Responses Moree a sit 
| $100 gift 


—— 


100 


50 
Attribute Choice Value 





Figure 1. Example of Method 1 (Step 2}, Marking a Linear Scale. 


Method 1 responses, considered by themselves, are 
inherently consistent and would become inconsistent only if the expert 
changes his rank ordering established in Step 1. 

c. Step 3: Aftribute Choice Pairwise Comparisons (Method 2) 

The second method by which the expert is asked to quantify 
the attribute choices, designated Method 2, is a process of pairwise 
comparisons between all attribute choices within each attribute. This 
method provides attribute choice values indirectly via ratios of attribute 
choice values. The expert is asked to give his opinion on the relative 
value of how much better (more valuable, more preferred, etc.) each 
best attribute choice is over the attribute choice with the second best 
level of the atiribute, third best, and so on down fo the attribute choice 


with the worst level of the attribute, as ranked during Step 1. He then 


1] 


compares the second best to the third best and so on down to the worst. 
This process is continued uniil all possible pairwise comparison 
combinations of attribute choices of each attribute are considered. The 


expression for the pairwise comparison ratio is shown in Equation (3): 


Vv | 

r= 7 = ratio of the values of attribute choices k and |, where (3) 
V, = value of attribute choice k, and 

V, = value of attribute choice |, and 


V.>V,. 


If K, is the number of attribute choices within attribute i, then the required 
number of comparisons Is Tae = —— 

Figure 2 demonstrates an example set of pairwise 
comparisons (Step 3, Method 2) with the three attribute choices for the 
attribute gift stature discussed earlier. 

A technique for checking the expert's Method 2 response 
consistency starts by plotting data on an (X, Y) plot. Letr,;, r,,andr,, be 
the expert's responses for the pairwise comparison ratios of the best versus 
intermediate, intermediate versus worst, and best versus worst attribute 
choices, respectively. Now plot the expert's response forr,,, on ihe 
X-coordinate. From Equation (3) we know f,. = 2, andr, = — The 
product of these two ratios reduces to a derived value for rand Is 
plotted on the Y-coordinate. If the expert is perfectly consistent in his 


responses, [if = (“2 (#) = = =r, ,and the plotted data point will 


lie on a line through the origin of slope 1.0. A quick visual consistency 











check then notes the displacement, if any, of the plotted data point from 


the line drawn through the origin with slope 1.0. 


Give your quantitative opinions on how much bef?ter the best attribute 

| choice is over the intermediate and worst attribute choices, and how 
much better the intermediate attribute choice is over the worst. 
eAtiribute: Gift Stature 
e Attribute Choices: Best:.$100 gift. Intermediate:_$20 gift. Worst:_$5 gift 


Expert's responses... Ratio of best to intermediate: r,, = V,/ V, 


A 


®The (Best) $100 giftis_5 times better (more valuable) than the (intermediate) $20 gift. 


®The (Best) $100 gift is_19__ times beffer (more valuable) than the (Worst) $5 gift. 


®The (intermediate) $20 gift is_3.5 times better (more valuable) than the (Worst) $5 gift. 





Figure 2. Example of Method 2 (Step 3), Attribute Choice Pairwise 


Comparisons. 


Quantitative measures of response consistency involve 
calculating the vertical bias and mean squared difference of the data 
points from the slope 1.0 line. Equations for calculating these measures 
are discussed in Chapter IV, Section A. Figure 3 shows a consistency 
check plot of rr, versus r,, for the example data in Figure 2. 

d. Step 4: Attribute Choice Indifference Probabilities 

(Method 3) 
Method 3 (Step 4) is the third and final method used to 


quantify the expert's attribute choice preferences, and involves acquiring 


point from line, squared. In this 
example, mean squared 
difference is (17.5 -19.0)? = 2.25. 


Vertical Bias 

Vertical displacement above (+) or 
below (-) the line. In this example, 
vertical bias is 17.5 - 19.0 = -1.5. 
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Figure 3. Checking Consistency of Method 2 Responses. 


the expert's attribute choice indifference probability. Perhaps the easiest 
way to define an indifference probability is with a decision tree. 

| Figure 4 shows a simple decision tree, or decision sapling 
[Ref. 1]. The square node on the left side of the diagram is a decision 
node, the round one is a chance node, and the diamond shaped nodes 
are resulf nodes. The result node labels V,, V,, and V,, are the values of 
the best, intermediate, and worst attribute choices, respectively, for a 
given attribute. The decision maker starts at the decision node and 
moves from left to right along one and only one path until he reaches a 
result node. There is no risk involved for the expert if he chooses to follow 
the upper branch. Upon reaching the result node he will receive a value 


V,. He may, on the other hand, choose to take the lower branch. Upon 
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Figure 4. Sample of a Decision Sapling. 










reaching the chance node, there is a chance, p, he will take the upper 
branch out of the chance node and receive a value V,, or a chance 
(1 - p) that he will follow the lower branch and receives a value V,. The 
indifference probability is the value of p for which the decision maker is 
indifferent between taking the gamble and taking the riskless alternative. 
Figure 5 shows an example of a type of problem that could 
be posed to an expert to determine his indifference probability. The 
example uses the same attribute (gift stature) and attribute choices 
($100.00 gift (best); $20.00 gift (intermediate); and $5.00 gift (Worst)) as in 
previous examples. 
A technique for checking consistency within Method 3 
requires two responses for each attribute. The expert first provides a 


response to the original indifference probability problem as demonstrated 


1S 


eYou currently have the (intermediate) $20.00 gift. You may keep it, or you 
may use it to gamble for the (Best) $100.00 gift. If you gamble and win, you 
get the (Best) $100.00 gift. If you gamble and loose, you get the (Worst) $5.00 
gift. If you do not gamble at all, you keep the (intermediate) $20.00 gift. 


eWhat is the minimum percentage chance of winning the (Best) $100.00 gift 
you are willing to accept in order to take the gamble? 


Expert's Response, (p x 100) — 20 percent 


Figure 5. Sample Method 3 (Step 4) Problem to Induce Expert's 





Indifference Probability, p. 


in Figure 5. The expert then provides a second response to an 
indifference probability problem that could be considered the inverse of 
the original. The expert's response to the first indifference probability 
problem is plotted on the X-coordinate, while one minus the 
corresponding response from the second indifference probability problem 
is plotted on the Y-coordinate. Figure 6:demonstrates an example of a 
second attribute choice indifference probability problem corresponding 


to the first problem in Figure 53. 


eYou currently have the (intermediate) $20.00 gift. You may keep if, or you 
may use it to gamble for the (Best) $100.00 gift. If you gamble and win, you 
get the (Best) $100.00 gift. If you gamble and loose, you get the (worst) $5.00 
gift. If you do not gamble at all, you keep the (intermediate) $20.00 gif. 


eWhat is the maximum percentage chance of getting the (worst) $5.00 gift 
you are willing to accept in order to take the gamble? 


Expert's Response, (q x 100} e 70 percent 


Figure 6. Sample Method 3 (Step 4) Problem to Induce Second Atiribute 





Choice Indifference Probability, a. 


16 | 


a Ln eR EG ee Lee eee ee ey ee eo 











The indifference probability equates the expected values out 
of either branch of the decision node in the decision sapling such that the 
exper is indifferent between taking the riskless branch or the risky venture. 
When | 


= pV. + (1-p)V,, = where p= Pr{win}, | (4) 


V; 
V, =(l-q)V, + qV,, where q= Pr{loose}, and (5) 


Vi.V,,V, = values of best, intermediate and worst 


attribute choices, respecitully, 


the expert is perfectly consistent. Figure 7 shows a consistency plot for the 


responses given in the examples in Figures 5 and 6. Note that if the expert 


(X, Y) Values of plotted 
point = (p, 1-q) = (0.2, 0.3). 


Line of Slope 1.0 


Mean Squared Difference 
Vertical displacement of data point from 
line, squared. In this example, mean 
squared difference is (0.3 - 0.2)? = .01. 


Vertical Bias 

Vertical displacement above (+) or below 
(-) the line. In this example, vertical bias is 
0.3 - 0.2 = +0.1. 
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indifference Probability, p 


Figure 7. Method 3 (Step 4) Indifference Probability Intra-Method 


Consistency Check. 





is perfectly consistent with his responses, then 


p = (1 -q). | (6) 


2. Inter-Method Consistency Checks 
Inter-method consistency check procedures are presented in the 
following subsections. As with the intra-method consistency checks, these 
checks involve plotting data on an (Xx, Y) plot. If the expert is perfectly 
consistent, a line drawn from the origin through the (X, Y) data point 
should have a slope of 1.0. 
a. Consistency Check Between Methods I and 2 
The first inter-method consistency check compares the 
expert's responses to Method 1 (marking a linear scale) and Method 2 
(attribute choice pairwise comparisons). The expert's Method 2 responses, 
the r,'s, are plotted on the X-coordinate. The corresponding calculated 
ratios of values from Method | are plotted on the Y-coordinate. For 
example, from Figure 2, the expert's response for the best versus worst 


pairwise comparison was 19. From Figure 1, the ratio of the best attribute 
Vb 


choice value, V,, versus the worst attribute choice value, V,,, was 7 = 
24 = 18. If the expert was perfectly consistent between methods the — 
plotted point should lie on a line through the origin of slope 1.0. Figure 8 
shows the inter-method consistency check for the data from the figures 
mentioned above. 

b. Consistency Check Between Methods I and 3 

A method for checking consistency between Methods | 
and 3 is now discussed.. The relationship between p and the ratio of value 
differences is first derived. From Equation (4), 
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V. = pV, + (I-p)V,. 


=pV, - pv, + V,. 
(V, -V,) = plV, - Vy). 
(Vi-Vw) 





Ins Sas (7) 
Each p is plotted on the X-coordinate and the corresponding — on 


the Y-coordinate. Again, if the expert is perfectly consistent between 
methods, the plotted points will lie directly on a line through the origin of 


slope 1.0. 


(X%, Y) Values of plotied 
points = (r,,, V/V): (3.5, 4.0), 
(5.0, 4.5), (19.0, 18.0). 


7 
Mean Squared Difference 
Sum of squared data point vertical 
displacement from the line, divided by the 


<a squared difference is: 


sf ee Be — Pf [(4.0-3.5)? + (4.5-5.0)? + (18.0-19.0)? } / 3 = 0.50. 
Bes tt | Vertical Bias 


Sum of vertical displacements above (+) or 
below [-) the line, divided by the number of 
data points. In this example, vertical bias is: 

(4.0-3.5) + (4.5-5.0) + (18.0-19.0) = -1.0. 
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Figure 8. Inter-Method Consistency Check between Methods and 2, 


Marking a Linear Scale versus Attribute Choice Pairwise 


Comparisons. 





Figure 9 shows a consistency plot of the Methods | and 3 
sample data from Figures 1 and 5. Note that since probabilities are being 
plotted, the scales on this consistency plot are values between 0.0 and 
1.0, unlike the previous inter-method consistency plot which Is scaled by 


ratios of attribute values. 


. (X, Y) Values of plotted point = 
(p, (V,- VJ/(V, - V)) = (0.176, 0.2). 


Mean Squared Difference 
Vertical displacement of data point from 
line, squared. In this example, mean 
squared difference is (0.176 - 0.2)* = 0.0006. 
Vertical Bias 
Vertical displacement above (+) or below 
(-) the line. in this example, vertical bias is 
0.176 - 0.2 = -0.024. 
Line of Slope 1.0 
0 0 0.2 0. 
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Figure 9. Inter-Method Consistency Check between Methods | and 3, 
Marking a Linear Scale versus Attribute Choice Indifference 


Probabilities. 


c. Consistency Check Between Methods 2 and 3 
The third and final inter-method consistency check compares 


the expert's responses between Methods 2 and 3. From Equation (7), 
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Y-coordinate. A consistency plot of the Methods | and 3 sample data 





The value of p is plotted on the X-coordinate, and the ratio on the 


would look very similar to Figure 9. 


This chapter provided definitions of multi-attrinute decision 
alternative sets, attributes, and attribute choices, and an overview of how 
these elements are combined to evaluate decision alternative sets. The 
employment of subjective inputs from subject matter experts was 
discussed, as were three methods for evaluating attribute choices. A 
presentation of the techniques used to apply the methods, along with 
ways to conduct intra and inter-method consistency checks, concluded 
the chapter. 

Chapter Ill goes into detail on the experimental approach 


utilized for this thesis. 
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Ii. EXPERIMENTAL APPROACH 


Chapter Ill highlights the experimental approach Used to collect 
the thesis data. Also included is a brief discussion of the history and 
current mission applications of LAMPS. The importance of and reasoning 
behind presenting the pilots with a standardized, albeit generalized, 
mission are also discussed. The specific attributes and attribute choices 
used for the experiment are then presented, along with samples of the 81 
possible helicopter system suites that could be created. The chapter 
finishes with an overview of the survey response form and guide, and a 


discussion on the actual administration of the surveys. 


A. UNITED STATES NAVY LAMPS HELICOPTER PILOT SUBJECT MATTER EXPERT 
DECISION MAKERS 

United States Navy LAMPS (Light Airborne Multi-Purpose System)’ 
helicopter pilots were selected to be the subject matter experts (i.e., 
decision makers) for an experimental application of the methods 
described in Chapter II due to the author's extensive experience and 
familiarity with the LAMPS community. A total of 27 LAMPS pilots assigned 
to the Naval Postgraduate School were individually administered a survey 
to solicit their expert judgements on the value of three pieces of 
helicopter equipment (attribute choices) in each of four different 
equipment categories (attributes). The survey also queried the subject 
matter experts to solicit data for developing equipment category 


weighting factors (atiribute weighting factors) for possible future analysis. 


Both U. S$. Navy LAMPS Mk | and LAMPS Mk III helicopter pilots were surveyed for 


this thesis. LAMPS Mk | pilots operate the SH-2F/G Seasprite, while LAMPS Mk Il! pilots 
operate the SH-60B Seahawk. 
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Data processing and analysis of the experts’ responses, especially 
response consistency checks, are presented in Chapters IV and V. 
The reader is reminded that it was not the intent of this thesis to 
develop values or weighting factors for LAMPS equipment, systems, or 
system suites, rather it was to exercise and check for expert response 


consistency of the methods presented in Chapter Il. 


B. WHAT IS LAMPS? 

LAMPS' ancestor was conceived in the late 1950's in the form of 
DASH (Drone Anti-Submarine Helicopter). DASH was an unmanned, radio 
controlled, torpedo carrying helicopter designed to deliver USW 
(UnderSea Warfare) weapons at convergence zone ranges, well beyond 
the five mile range of the parent ship. Deployed from USW ships, DASH 
was originally semi- expendable, which is just as well as few DASH's lived 
beyond a few flights. 

As torpedo size increased, so did the payload requirements, size 
and costs of DASH. By the late 1960's, DASH’s costs no longer justified 
expendability. By this time, helicopter and powerplant technology had 
matured to the point where manned helicopters could be deployed from 
surface combatants. In 1971, LAMPS Mk | was born in the form of the 
SH-2F Seasprite. In its infancy, LAMPS Mk | was a marriage of off-the-shelf 
sensor and avionic systems with previously boneyarded airframes. ‘It was 
an enormously successful relationship and the Navy quickly recognized 
the synergistically enhanced capabilities of a ship-helicopter system. 

In 1975, IBM won the System Integration Program contract to design 
LAMPS Mk Ill from the ground up. The LAMPS Mk III SH-60B Seahawk first 
deployed in 1984 and is the airborne component of a fully-integrated 
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ship-helicopter weapon system. The helicopter is crewed by a pilot, ATO 
(Airborne Tactics Officer), and an enlisted sensor operator (AW rating). It is 
highly capable of autonomous operations with all of its sensors and 
weapons, including radar, sonobuoys, MAD (Magnetic Anomaly 
Detector), IFF (Identification Friend or Foe), ESM (Electronic Support 
Measures), torpedoes, and Penguin Missiles. 

The helicopter is only half the story, however, as many of the 
airborne sensors' capabilities are greatly enhanced through a dedicated 
datalink to a LAMPS Team embarked in the parent ship. The ship's greater 
data processing and analysis capabilities, especially for sonar and ESM 
data, along with greater C*l (Command, Control, Communications and 
Intelligence) capability, make for a truly synergistic system. 

LAMPS was first cast primarily as an USW platform. Today LAMPS' 
two primary missions are ASUW (Anti-SUrface Warfare) and USW. The ever- 
growing list of secondary missions is expansive, and includes ASST 
(Anti-Ship Surveillance and Targeting), NSFS (Naval Surface Fire Support), 
SAR (Search And Rescue), VERTREP (VERTical REPlenishment), and 
MEDEVAC (MEDical EVACuation), to name a few. 

Currently the LAMPS Mk | is being phased out of the fleet, while the 
Mk Ill is deployed extensively, embarked in Ticonderoga Class Cruisers 
(CG-47), Soruance (DD-963) and future Arleigh Burke (DDG-51)} Class 
Destroyers, and Oliver Hazzard Perry Class Frigates (FFG-7). [Refs. 2 - 6] 


C. STANDARDIZED MISSION 

In view of the wide range and variety of missions performed by 
LAMPS aircrews, and the numerous significant operational and tactical 
considerations that dictate which helicopter systems to use and how to 
25 





employ them given a specific mission, a standardized mission was 
presented to the LAMPS pilots from which to reference their responses. 
The standardized mission presented to the LAMPS pilots was a: 
generic USW mission”. Before taking the survey, pilots were briefed to tailor 
their responses in reference to a complete USW mission; i.e., from the 
redetection phase all the way through to completion of the battle 
damage assessment (BDA) phase. Pilots were repeatedly reminded 
throughout the survey to reference their responses to a complete USW 
mission. Pilot response in reference to a generalized USW mission was 
selected over presenting a more detailed mission scenario to avoid 
possible complications caused by information overload. It was felt that 
giving further mission scenario details (e.g., target type (nuclear or diesel 
submarine), weather conditions, water characteristics (depth, 
temperature gradients, salinity, biologics, etc.}, area (littoral or blue 


water}, etc.), would generate more confusion than clarification. 


D. EXPERIMENT ATTRIBUTES, ATTRIBUTE CHOICES, AND MULTI-ATTRIBUTE 
DECISION ALTERNATIVE SETS 

Four LAMPS helicopter system categories vital to the successful 
execution of a complete USW mission were presented to the pilots. The 
four system categories, weapons, navigation systems, communication 
systems, and sensors, made up the attributes for the methods described in 
Chapter Il. Each of these attributes was comprised of three different 
system options (i.e., the attribute choices). Table | lists the attributes and 


corresponding attribute choices used in the survey. 





: USW is a relatively new term to represent what historically has been called 
Anti-Submarine Warfare or ASW. 
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Attribute Mk-46 Global Datalink Magnetic 
Choices Torpedo | Positioning System (D/L) Anomaly 
(abbreviation)|| = (Mk-46) (GPS) Detector (MAD) 

pede 500# Depth Tactical Air Satellite Surface 
implied i“ Charge Navigation Communications | Search Radar 
table ordering (500#) (TACAN) (Sat Com) (Radar) 
of attribute Mk-50 Doppler UHF/VHF Forward Looking 
choices.) Torpedo Radar Radio Infrared 





(Mk-50) (Dop) 


Table 1. Survey Attributes and Attribute Choices. 


{Radio} (FLIR) 





Given the LAMPS system categories and individual systems (i.e., the 
attributes and attribute choices), LAMPS helicopter USW mission system 
suites (i.e., alternative sets) could be formed. Any system suite would be 
comprised of one individual system option from each of the four system 
categories. Given four attributes and three attribute choices per 
category, there would be j = 81 ( 3* = 81 ) possible LAMPS helicopter 
mission system suites for the pilots to rank. For example: 

j= 1; (Mk-46, GPS, D/L, MAD), 

| = 2; (Mk-46, GPS, D/L, Radar}, 

j = 3; (Mk-46, GPS, D/L, FLIR), 

j = 4; (Mk-46, GPS, Sat Com, MAD), 


j = 81; (Mk-50, Dop, Radio, FLIR). 
7 








E. DATA COLLECTION SURVEY 

Copies of the survey instruction guide and survey response form 
used to collect data are in Appendix B. Each survey was individually 
administered with the author readily available to answer questions 
regarding the mechanics or semantics of the survey. The author made 
every effort to not plant response predispositions in pilots taking the 
survey. Pilots were not told until after completing the survey that their 
responses would be examined primarily for consistency as opposed To 
trying to quantify the value of a LAMPS system suite. Most of the 27 pilots 
taking the survey completed it in 45 + 10 minutes. With few exceptions 


the survey follows the order of steps described in Chapter Il. 


This chapter presented the experimental approach used to collect 
data for the thesis, as well as brief discussions of the history and current 
mission applications of LAMPS. The importance of and reasoning behind 
presenting the pilots with a standardized mission were also discussed. The 
specific attributes and attribute choices used for the experiment were 
presented, along with samples of the 81 possible helicopter system suites 
that could be created. The chapter closed with an overview of the 
survey response form and guide, and a discussion on the actual 
administration of the surveys. 

Chapter IV presents the methods used to analyze the data, 


specifically intra and inter-method consistency checks. 
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IV. CONSISTENCY WITHIN AND BETWEEN METHODS 


This chapter describes the methods used to analyze the data, 
specifically intra and inter-method consistency checks. Consistency 
check plots, along with quantitative consistency measures, are presented 


for five main data subsets. 


A. CONSISTENCY CHECKS OVERVIEW 

Five sets of consistency checks are discussed in this chapter, two 
intra-method and three inter-method. Each set of checks employs three 
techniques for analyzing the LAMPS pilots’ survey response consistencies. 
The first technique involves plotting corresponding data subsets on an 
(X, Y) plot along with a line through the origin of slope 1.0. If the 
experimental responses are perfectly consistent, all the data points lie on 
the line. The reader can see in Figures 10 through 14 in the subsequent 
sections of this chapter that while some of the plots visually indicate a 
moderate degree of consistency, others appear to show a great deal of 
inconsistency. 

The second consistency check technique examines the square root 


of the mean squared difference, or dispersion parameter, {a2 =d,of the 
data points’ vertical displacement from a line through the origin of slope 


1.0. The magnitude of d gives a measure of the degree of consistency. 
The value of d will be zero if and only if all responses are perfectly 
consistent. Otherwise d will have a positive value. Equation (9) gives the 


formulation for the mean squared difference, d*: 


d? = LL, (ye -Xk)) ~~, Where (9) 
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d? = mean squared difference, 
K = the number of data points in the subset, 
x, = the X-coordinate of the k" data point, and 


y, = the Y-coordinate of the k" data point. 


Suppose the X-coordinate is plotted over a range (0, m), so a 
consistent point with x = m would be (m, m). The scale can be converted 
to (0, 1) by dividing each data point by m, so if d?___ is defined to be the 
normalized mean squared difference, from Equation (9), 


2 al yk (Ye _ %y2_ de 
CD? om = K Uk=I(m — m) m2 / 


and the normalized dispersion parameter is 


d =. (10) 


The units for the graphs shown in Figures 11, 13, and 14 are 
indifference probabilities and range in magnitude from zero to one. The 
units for the graphs shown in Figures 10 and 12 are ratios of attribute 
choice values. The X-coordinate axes for these graphs are limited to 10.0 
and the Y-coordinate axes to 20.0. In order to make quantitative 
comparative statements between consistency checks of different units, 
the d's for the checks involving attribute choice value ratios were 
normalized to a (0, 1) scale. This was done by dividing d by m = 10, the 
maximum value of the X-coordinate axis. After normalizing the ratio unit 
data to the same scale as the probability unit data, the d,,,,,'5 from all the 


data subsets can be compared direcily. 
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The final technique used to examine response consistency 


calculates the normalized vertical bias, b from a line through the origin 


of slope 1.0. The sign of b indicates to which side of the slope 1.0 line the 
data is biased. A positive b indicates bias above the line, negative | 


indicates below. The magnitude of b indicates the degree to which 


norm 


the data set is biased. Equation (11) gives the formulation for the 


normalized vertical bias, b 


norm* 


nom = =z Le (¥k — Xk)» Where (11) 
Dom = NOrmalized vertical bias, 
m = maximum valve for the X-axis, 
K = the number of data points in the subset, 
x, = the X-coordinate of the k" data point, and 


y, = the Y-coordinate of the k" data point. 


The value of m is 1.0 for the data subsets with indifference probability units, 
and 10.0 for the data subsets with attribute choice value ratio units. 
Appendix C is a tabulation of the raw survey data. Supplementary 


graphs of numerous data subsets are in Appendix D. 


B. EXPERIMENTAL INTRA-METHOD CONSISTENCY CHECKS 

The pilots’ responses fo Methods 2 and 3, attribute choice pairwise 
comparisons and indifference probabilities, were examined for 
intra-method consistency. 

1. Method 2: Attribute Choice Pairwise Comparisons 

Figure 10 presents a graph plotting the best versus worst pairwise 


comparison ratio, r,,, directly from the data as the X-coordinate, and the 
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Figure 10. Attribute Choice Pairwise Comparison Intra-Method 


Consistency Check, all Attridutes. 


product of the best versus intermediate ratio times the intermediate 


versus worst ratio, r,.1,,, as the corresponding Y-coordinate. This graph 


presents all the attribute choice pairwise comparison data, within the limits 
of the graph, from all four attributes (weapons, navigation systems, 
communication systems, and sensors). Visual inspection of the graph 
reveals the derived best versus worst ratios, the Y-coordinates, are 
somewhat consistent with the ratios obtained directly, and are definitely 


biased towards the derived ratios. The plotting method used does not 
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indicate multiple points in the same location, so it is difficult to get a true 
visual picture of the amount of consistency or bias. The values of d_._ 
and b,.,, are 0.0299 and +0.1401, respectively, indicating a high level of 
consistency, but a moderate level of bias in the answers. Supplementary 
graphs displaying the data for each of the four attributes are found in 
Appendix D. | 

2. Method 3: Attribute Choice Indifference Probabilities 

Figure 11 presents a plot of the indifference probability, p, directly 
from the data on the X-coordinate, and the corresponding one minus the 
second indifference probability, gq, as the Y-coordinate. This graph 


presents all the attribute choice indifference probability data, both first 


One Minus "Second" Indifference Probability, 1-q 
Weapons, Navigation, Communications, Sensors 


0. 
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Indifference Probability, p 
Weapons, Navigation, Communications, Sensors 


Figure 11. Attribute Choice Indifference Probabilities Intra-Method 
Consistency Check, all Attributes. 
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second, from all four attributes. Visual inspection of the graph indicates a 
less consistent set of responses than with the pairwise comparison data of 
the previous graph, but a much smaller positive bias as the data is much 
more evenly spread both above and below the line. For these 
observations d_. = 0.1851 and b,,,,,, = +0.0420. Supplementary graphs 


displaying the data for each of the four attributes are found in 


Appendix D. 


C. EXPERIMENTAL INTER-METHOD CONSISTENCY CHECKS 

The pilots’ responses fo all three methods, marking a linear scale, 
attribute choice pairwise comparisons, and indifference probabilities, 
were examined for inter-method consistency. 

1. Between Methods 1 and 2: Marking a Linear Scale versus 

Attribute Choice Pairwise Comparisons 

Figure 12 presents a plot of pairwise comparison data on the 
X-coordinate, and corresponding ratios of linear scale values from 
Method 2 on the Y-coordinate (see Equation (3)). The degree of 
consistency between Methods 1 and 2 at first appears to be notably less 
than the degree of consistency within either Methods 1 or 2. A second 
characteristic difference from the intra-method checks is the vertical 
grouping of the data poinis. Intuitively this is explained by the pilots 
propensity for integer values when providing ratios of attribute choice 
values inputs. Over 91 percent of the points on the x-coordinate are 
integer value. 

The value of d___ for this data is only 0.0361, almost the same as 
the best intra-method value. This result seems to contradict what is 
displayed graphically, until the data is examined more closely. Over 7? 
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percent of the X-coordinate values are five or less, and nearly 40 percent 
are two or less (out of 286 data points). The relatively small value for d_._ 
is then somewhat explained by the much larger volume of data points in 
the groupings in the lower left corner of the graph and close to the 
slope 1.0 line. | 

Finally, and not surprisingly looking at the plot, this check was 


strongly negatively biased with a b of (-0.2264)}. Supplementary 


graphs displaying the data for each of the four attributes, and for the 
three types of attribute choice pairwise comparisons, are found in 


Appendix D. 
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Figure 12. Marking a Linear Scale versus Atiribute Choice Pairwise 


Comparisons Inter-Method Consistency Check, all Attributes. 
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2. Between Methods 1 and 3: Marking a Linear Scale versus 
Attribute Choice Indifference Probabilities 

Figure 13 presents a plot of the indifference probabilities directly 
obtained from Method 3 on the X-coordinate. The corresponding derived 
indifference probabilities, from Method 1 data, is plotted on the 
Y-coordinate (see Equation (7}). This graph also shows some pilot 
disposition for vertical grouping of data points, though not quite as much 
as in the previous graph. The raw data for the X-coordinates are 


percentages. Only three of the 108 X-coordinate inputs are integer 
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Figure 13. Marking a Linear Scale versus Attribute Choice Indifference 


Probabilities Inter-Method Consistency Check, all Attributes. 
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values not divisible by five, and only 30 of the 108 are not divisible by ten. 
Reviewing the data helps explain why the vast majority of data points fall 
on the vertical gridlines. - 

The value of d_.,, for this check is 0.2968, significantly greater than 


the three previous consistency check results. Reviewing the graph 


confirms the large d___ value, seemingly indicating a relatively low level 


of inter-method consistency between Methods 1] and 3. 

Vertical bias is relatively small, as might be expected by the 
considerable and apparently evenly distributed vertical displacement of 
the data points on either side of the slope 1.0 line. The value of DL orm IS 
(-0.0862), the least biased of the inter-method consistency checks 
discussed in this chapter. Supplementary graphs displaying the data for 
each of the four attributes are in Appendix D. 

3. Between Methods 2 and 3: Attribute Choice Pairwise 

Comparisons versus Attribute Choice Indifference Probabilities 

The final inter-method consistency check compares directly 
obtained indifference probabilities and corresponding indifference 
probabilities derived from the attribute choice pairwise comparison 
method data (see Equation (8)). Figure 14 presents a plot of the 
directly-obtained indifference probabilities on the X-coordinate and the 
corresponding derived values on the Y-coordinate. Since this graph has 
the same inputs for the X-coordinates as the previous graph, it also shows 
correlation along the vertical gridlines. The graph indicates a rather low 
level of consistency and a strongly negative bias. 

The value of d.. for this check is 0.3962, the largest d_.,,, value of 
the five consistency checks discussed in this chapter, and greater than all 
but one of the data subset checks found in Appendix D. Such a large 
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value of d_. suggests a significant level of response inconsistency 


between Methods 2 and 3. 

The graphical interpretation of the bias is supported by a onan of 
(-0.2052), indicating a strongly negative bias. Supplementary graphs 
displaying the data for each of the four attributes are found in 


Appenaix D. 
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Figure 14. Inter-Method Consistency Check, Attribute Choice Pairwise 


Comparisons versus Indifference Probabilities, all Attributes. 


38 








D. SUMMARY OF CONSISTENCY CHECK RESULTS 

Table 2 summarizes the d’, d, d_._, b.., and number of data points, 
K, for each of the consistency checks discussed in this chapter. Table 3 
summarizes the same information for the data subset consistency check 


graphs found in Appendix D. 










































Method 2: 8.9314} 2.9885] 0.0299] +©.1401 
Pairwise Comparisons (10) 
0. 0343 0.1851} 0. 1851 ver 0420] 108 

indifference Probabilities (11) 
Entire data set, all attributes 
Methods 1 and 2: 12.968} 3.601] 0.0361] -0.2264) 286 
Marking a Linear Scale vs Pairwise 
Entire data set (W/in graph limits), all attributes _ 
Methods 1 and 3: 0. 0881 0. 2968 0.2968} -0.0862] 108 
Marking a Linear Scale vs Indifference 

Probabilities (13) 
Entire data set, all attributes _ 

Probabilities (14) 
Entire data set, all attributes 


Consistency Check (Figure Number) 
Intra-method Consistency Checks aS ti | Pana 
Entire data set (w/in graph limits), all attributes 
Method 3: 
inter-method Consistency Checks dd | dtd |b. | K. 
_ Comparisons (12) 
Methods 2 and 3: 0.1569} 0.3962) 0.3962] -0.2052] 103 
Pairwise Comparisons vs Indifference 
Table 2. Summary Table of Consistency Check Parameters for Data 


Subsets Presented in Chapter IV. 
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Consistency Check (Figure Number) aaa 
Intra-method Consistency Checks | d | d [di b |K. 


Method 2: Pairwise Comparisons | 8.6406] 2.9395 0.0294] +0.1538]. 20 
Weapons (A.D-1) | 
Navigation (A.D-2) 5.4167| 2.3274] 0.0233} +0.1250| 12 


Method 3: indifference Probabilities 
Weapons (A.D-5) 


Methods 1 and 2: 13.077] 3.61621 0.0362 "T 


Marking Linear Scale vs Pairwise Comparisons 
Best vs Intermediate (A.D-9) 

Best vs Worst (A.D-10) | 18.641] 4.3176] 0.0432} -0.3008] 80} 
| 13.341] 3.6526] 0.0365] -0.2376) 69; 


Methods 1 and 3: Marking Linear Scale vs_ § 0.0759| 0.2754| 0.2754| -0.1556| 27 
indifference Probabilities 

Weapons [A.D-16) | 

Navigation (A.D-17) 0.0995| 0.3155] 0.3155] -0.1513| 27 


Methods 2.and 3: Pairwise Comparisons vs_ | 0.1316! 0.3627| 0.3627| -0.2761 


Indifference Probabilities 
Weapons [A.D-20) | 


Table 3. Summary Table of Consistency Check Parameters for Data 























Subsets Presented in Appendix D. 


AO 











This chapter described the methods used to analyze the data, 
specifically intra and inter-method consistency checks. Consistency 
check plots and quantitative measures of consistency were presented for 
five main data subsets. The chapter finished with tables summarizing the 
consistency check parameters for 28 various data subsets. 

The final chapter will interpret the consistency check parameters 


developed in Chapter IV for the five main data subsets. 


4] 
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V. SUMMARY AND CONCLUSIONS 


This chapter provides interpretations of the consistency check 
parameters developed in Chapter IV for the five main data subsets. 
some consistency checks that yielded unexpected results are analyzed in 
greater detail and utility function theory applied in an attempt to explain 
the results. The thesis closes with discussions on applicability of the 


methods, experiment conclusions and suggested areas for future study. 


A. EXPERIMENT RESULTS 

The following is a discussion of the experiment results. Table 4 is a 
reduced version of Table 2 and presents the primary consistency check 
, and the 


for the two intra-method and three inter-method 


parameters, the normalized dispersion parameter, d 


normalized bias, b, 
consistency checks encompassing ail attributes. 
1. Intra-method Consistency Checks 
a. Within Method 2 


The dispersion consistency check parameter, d for 


Method 2 was only 0.0299, indicating a low level of data spread within this 
method. 

Expert responses to this method were moderately positively 
biased, with b_ = +0.1401. From the discussion in Chapter Il, subsection 
D.1.c, and Equation (3), we know that if the experts were perfectly 


consistent, then 


low = [biliw- 
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Consistency Check (Figure number) 






































Intra-method Consistency Checks fd | bo 
Method 2: 0.0299; +O.1401 
Pairwise Comparisons (10) 
Method 3: | 0.1851} +O.0420 
Indifference Probabilities (11) 
Entire data set, all attributes 
Methods 1 and 2: 0.0361) -0.2264 
Marking a Linear Scale vs Pairwise Comparisons (12) 
Entire data set (w/in graph limits) , all attributes 
0.2968) -0.0862 
0.3962} -0.2052 


Entire data set {w/in graph limits), all attributes 
Inter-method Consistency Checks i ee ee 
Table 4. Summary of Primary Consistency Check Parameters. 









Methods 1 and 3: 
Marking a Linear Scale vs Indifference Probabilities (13) 
Entire data set, all attributes 


Methods 2 and 3: 
Pairwise Comparisons vs Indifference Probabilities (14) 
Entire data set, all attributes 












A positive bias indicates that, on average, experts overestimated the 
derived values forr,, relative to the directly obtained values. 

b. Within Method 3 

Consistency check parameter magnitudes for Method 3 
were the converse of those for Method 2. The d_,, value was relatively 
large at 0.1851, while a nearly negligible level of bias was indicated with a 
bm Of only +0.0420. Figure 15 replots the data for Method 3 with p 
plotted on the horizontal axis and p - (1 - q) plotted on the vertical axis. 
For this plot, the horizontal axis effectively displays the line of slope 1.0 


from Figure 11. This presentation confirms the consistency parameters 
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Note: The plotting method used 
for Figure 15 does not indicate 
multiple points with the same 
location. 


indifference Probability, p 


Figure 15. Alternate Intra-Method 3 Check: p - (1 - q) versus p. 


analysis that the derived values for p, the (1 - q)'s, are rather widely, but 
fairly evenly dispersed on either side of p. 
Figure 16 pictorially demonstrates Equation (6), i.e., ine two 


decision saplings are equivalent if and only if 


pb =(1-q),andq=(1-p), where 0<p,q<l. 





Figure 16. Two Equivalent Decision Saplings. 
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The parameter values indicate the experts may have had 
difficulty translating a corresponding value of g fora given p. This 
conclusion is supported by the fact that survey steps 4(a) and (b) (see 
Appendix B) used to collect data inputs for p and g, generated the 
largest number of clarification requests from the experts while they were 
taking the survey. A supporting interpretation of these results is that while 
marking a linear scale or conducting pairwise comparisons were rather 
natural actions for the experts, the experts were not practiced nor familiar 
with the concept of indifference probabilities as a tool for making 
comparisons. Expert training and practice with indifference probabilities 
might very well reduce the response dispersion for this method, improving 
the level of consistency. 

2. Inter-method Consistency Checks 

a. Between Methods I and 2 

The d_.., value for the check between Methods 1 and 2 was 
only 0.0361, indicating a relatively high level of response consistency 
between these two methods. The inter-method bias, however, was 
substantial and strongly negative, with b_._ = (-0.2264). A negative bias 
indicates that, on average, the derived attribute value ratios were 
understated relative to the directly obtained ratios. The relatively large 
magnitude of the bias suggests the possibility of a systematic prejudice in 
the method, possibly induced by the linear scale presentation. Reviewing 


Equation (3), if the consistency relationship is negatively biased, then, on 


average 


Vb Vb Vj 
bj > VV," Tow > Vw! ana liw > Vw! where Vinumerator 2 V denominator > 0. 
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Given the above, if a constant inter-Method 1 and 2 
correction factor, c,,, such thatO<c,,<v 
both v 


is subtracted from 


denominator ’” 


and v 


numerator denominator 


; then the ratio (teense increases in 
deno min ator 

magnitude. Thus, there is a value of c,, for each data point such that 

when subtracted from the values in both the numerator and 

denominator, the ratio of values equals the directly obtained ratio. The 

effect, then, of c,, for a negatively biased relationship between Methods 

1 and 2, is to effectively shift the origin of the linear scale to the right. For 


example, ifr, = 5, V, = 20 and V, = 60, then 


V 
Mow = 5 > S = = 72 ,and 
Vp-c 60-—c 
(8 — pa? = 2 => C19=10 








Vw-C19 = 20-C 19 


Figure 17 pictorially demonstrates the effect of the c,, = 10 for 
the r,, V,, and V,, in the example. The strong negative bias suggests the 
experts had a propensity for the high end of the linear scale, and/or an 


aversion to the low end. 





Figure 17. Pictorial Representation of Correction Factor c,,. 
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If b... > 0, then a constant c,, is added to both the 
numerator and denominator. This effectively shifts the origin of the linear 
scale to the left. Application of a correction factor to the entire data 
subset might therefore be used to reduce systematic bias in the experts: 
responses due to a propensity for either end of the linear scale. 

Finding a globally optimal value of c,, directly is difficult 
because of the existence of local minima and maxima. Heuristic and 
iterative approaches to find a near optimal c,,, however, are relatively 
easy to perform. To test the concept of a correction factor, integer 
valued c,,'s from one to ten were iteratively applied to the experimental 


data and the effects onb.. noted. The magnitude of b,.,,, Was 


reduced from an uncorrected value of (-0.2264}, to a negligible value of 
0.0241 when c,, = 9, a tenfold improvement. 

b. Between Methods I and 3 

The value of d___ for the check between Methods | and 3 
was 0.2968, which seems to indicate a low level of response consistency 
between these two methods. The modest b,,,,,, Of (-0.0862) was the 
smallest level of bias of the three inter-method consistency checks. To 
investigate the high value of d_.,. further, Figure 18 replots the data for this 
inter-method check with p plotted on horizontal axis, and p- a 
plotted on the vertical axis. As with the Method 3 intra-method 
consistency check, this presentation confirms the consistency parameters 
analysis that the derived values for p are widely, but fairly evenly 
dispersed on either side of p. An interesting and unanticipated 
phenomenon also surfaced in Figure 18. The bias, which is the derived 


minus the directly obtained indifference probability, appears to become 
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more negative with increasing p. This data behavior warranted closer 


examination to search for an explanation. 


Note: The plotting method used 
for Figure 18 does not indicate 
multiple points with the same 
location. 





indifference Probability, p 


Figure 18. Alternate Inter-Methods 1 and 3 Check: 
p-[(V,- VJ/(V, - Vj] versus p. 


From Equation (7), if the experts were perfectly consistent, 








then p= (yee) over the entire range of p and all the data points would 
lie on the horizontal axis. However, in Figure 18 it is evident that on 
average, 
if p>0.5, then p> (%), (12) 
: ViEVw 
if p< 0.5, then p< (wee) and (13) 
if p=0.5, then p=(**). (14) 
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Apparently there is a behavioral phenomenon that caused 


the experts to respond inconsistently. 


Consider the left decision sapling in Figure 16, except this time 


the result node outcomes are measured by option utility values, U(i), U(b), 


and U(w), instead of the options themselves, for the intermediate, best, 


and worst option utility values from top-to-bottom, respectively. Modifying 


Equation (4) by substituting V,, V,, and V, with U(i), U(b), and U(w), 


respectively, we get 
U(i) = pU(b) + {1 - p)U(wW}, thus 
p= (gor). 

The relationship between Equations (12) and (15) then is 
(Geaues) <P> GE). 


The relationship between Equations (13) and (15) then is 








a) _ &23 
[ woe = Ve-Vw/ ° 


(19) 


(16) 


(17) 


Figure 19 plots both a risk neutral utility function (line AB) and 


a generic risk averse utility function (curve CD). Let V,, V, and V,, plotted 


on the X-coordinate, be the values of the worst, intermediate, and best 


options, respectively. Let U(w), U(i), and U(b) be the utility values from the 
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risk averse utility function corresponding to the worst, intermediate, and 


best options, respectfully. Note that 


forall V,such that V,<V,<V,, then 








(wot) po (Yere) 48) 


Therefore, a person whose data values of p, V_, V, and V, support 


Equation (18) is displaying risk averse behavior. 


U (b) SS aE ae ae ee ee 


Ul) | 
AB: Risk Neutral Curve 


CD: Risk Averse Curve 
Vi Vi Ve: Values of worst, 
intermediate, and best options. 
U(k}: Utility of option k 
for risk averse curve. 
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Figure 19. Plot of Risk Neutral and Risk Averse Utility Function Curves. 


Similarly, if the utility function curve is convex, the inequality in 


Equation (18) reverses and 





(wore) =p < (Te) (19) 
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implying that data satisfying Equation (19) indicates the person is 
displaying risk prone behavior. 


From Equation (14), 





Vie \ 
p= (wove =0.5, 
V.-V, = 0.5(V, - V,) 
2(V,.-V)=V.,- Vy 


Vi = (Me), (20) 


2 
Similarly, from Equation (13), for p <0.5, 
Vi>PVpb+CU-p)Vw, and (21) 
from Equation (12), forp>0.5, 
Vi<PVp+(1-p)Vw. (22) 


The risk neutral, risk prone, and risk averse regions of Figure 18 
are those that satisfy Equations (20), (21), and (22), respectively. Figure 20 


displays the non-neutral regions pictorially. The experts, therefore, are risk 
averse when p is large, risk neutral when p = 0.5, and risk prone when p is 


a 


small. 
Equations (20) through (22) also make intuitive sense. 


Figure 21 shows a linear scale with V, and V, marked towards either end, 
and the risk prone, neutral, and averse regions labeled. If V, is at point 1, 


the intermediate option's value is not that much greater than that of the 
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Risk Averse Region 


Indifference Probability, p 





Figure 20. Risk Prone and Risk Averse Regions of Figure 18. 


worst option, and the expert is willing to accept a p that results in a 
negative expected value for agamble between the best and worst 
options, rather than accept the intermediate option for certain. 
Conversely, if V, is at point 2, the intermediate option's value is almost as 
great as the best option's value, and the expert must have a _p large 


enough to produce a positive expected value in the gamble. 


Risk Neutral 


Increasingly Increasingly 
Risk Prone Risk Averse 





Figure 21. Linear Scale Risk Prone, Neutral, and Averse Regions. 
a3 
























In summary, the plot in Figure 18 displays a distinctive pattern 
that was not anticipated. Analyzing the plot reveals that the experts were 
risk prone when V, was closer to V, than V,,, and risk averse when V, was 
closer to V, than V\,. 

c. Between Methods 2 and 3 

The inter-method consistency check between Methods 2 
and 3 displayed the same gross parameter behavior as with the 
preceding check. The value of d_._ for this check, the largest of the five 
main data subsets, was 0.3962, which appears to indicate a very low level 
of response consistency between these two methods. The inter-method 


bias was strongly negative at (-0.2052). Figure 22 replots the data for this 
inter-method check with p plotted on the horizontal axis, and p-“— 


(T bw!) 


plotted on the vertical axis. 


Pp - (Tiw - 1)/( bw - 1) 





Note: The plotting method used 
for Figure 22 does not indicate 

multiple points with the same 
location. 










indifference Probability, p 






Figure 22. Alternate Inter-Methods 2 and 3 Check: 
p-[(r - 1)/(r,y- 1)] versus p. 
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As with the inter-method consistency check between 
Methods 1 and 3, this check also displayed the curious behavior of the 
bias seemingly becoming more negative with increasing p. Similar 


arguments can be made to show that, on average, the experts were risk 
prone, neutral, and averse when p < 0.5, p = 0.5, and p > 0.5, respectively. 


B. METHOD APPLICABILITY 

Analysts using the methods described in and employed to conduct 
an experiment for this thesis could find applicability in numerous 
qualitative situations. The methods are not substitutes for thorough 
cost-benefit or similar analyses. However, in situations where there is a 
readily available pool of subject matter experts, but not an available, 
affordable, or computationally logical data base, these methods show 
definite applicability for developing a quantifiable and comparable 
measure of value, preference, worth, etc. 

Given several decision alternative sets, whether they are tanks, 
bicycles, or televisions, one could relatively quickly dissect the alternative 
sets into attributes and attribute choices. The survey question formats 
exercised for this thesis could then be adapted for the particular 


alternative sets under evaluation. 


C. EXPERIMENT CONCLUSIONS 
The procedures used in this thesis can be employed to 
quantitatively evaluate attribute choices with no natural measurement 
basis, the first major step in evaluating multi-atiripute decision alternatives. 
The intra-method consistency checks of the experimental data 


showed that the pairwise comparison method (Method 2} yielded notably 


30 





smaller dispersion parameters than the indifference probability method 
(Method 3). Method 2 responses, however, were much more biased than 
those of Method 3. These general observations imply there may be 
possible tradeoffs available between dispersion and bias between 
Methods 2 and 3. Consistency parameters for checks between methods 
were generally greater than the intra-method parameters. 

The most interesting observation borne out by variants of the 
consistency checks was not anticipated before conducting the checks. 
This was the incontestable relationship between expert response behavior 
and attribute choice and indifference probability values. The experts 
displayed risk prone behavior when they perceived there was little to 
loose and much to gain. Conversely, the experts displayed risk averse 


behavior when they perceived there was much to loose and little to gain. 


D. AREAS FOR FUTURE STUDY 

A considerable amount of time spent on this thesis was devoted to 
identifying the experiment subject matter, approach, and procedures, 
and survey development, trial testing, and modification. The time 
devoted to experimental approach and survey development preempted 
the thorough discussion and consistency analysis of attribute weighting 
factors. The first and foremost area of future study would be to develop 
the discussion and analysis of attribute weighting factors, analyze the 


remaining data in Appendix C, and develop actual alternative set values. 
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APPENDIX A. ATTRIBUTE, ATTRIBUTE CHOICE, AND MULTI-ATIRIBUTE 
DECISION ALTERNATIVE SET NOMENCLATURE AND INDEXING 


This appendix provides a detailed explanation of the nomenclature 
and indexing system used throughout this thesis. Equation (1) [see 
Chapter Il, Section B] is presented below as a springboard for the 


following discussion: 


V(j) = Lieve, V, (i) foranyj=1,2,...,m, where (1) 
V(j) = overall value for alternative set j; 
a, = weighting factor for attribute i; a, is the amount of set js 
overall value obtained from a unit of attribute I; 
a, > Oif, for atiribute i, bigger is better [Ref. 1, Table 7.2]; and, 
a, <Oif, for attribute i, smaller is better; and, 


V,(j) = value of attribute i for alternative set j. 


DEFINITIONS: 

¢Multi- Attribute Decision Alternative Set: Also called a decision 
alternative, alternative set, or sef, a multi-attribute decision alternative set 
is a complex object upon which a decision maker wants to place a 
quantitative value in order to directly compare it with similar complex 
objects. For example, an auto dealership may present a perspective car 
buyer (i.e., a decision maker) with dozens of alternative sets (i.e., cars) that 
he wanis to evaluate. 

eAttribute: An attribute is an alternative set characteristic common 
to all (or at least most) of the sets considered for evaluation and ranking. 


For example, attribute candidates for new cars could include gas 
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mileage and exterior color. Note that some attributes (e.g., gas mileage} 
are much easier to quantify than others (exterior color). 

eAttribute Choice: An attribute choice, or choice, is the particular 
item or characteristic of a certain attribute in each alternative set. For the 
car example, attribute choices for the exterior color attribute may be 
green, red, and black for some car models, and silver, dark blue, red, and 
turquoise for others. The gas mileage attribute choice for one model of 
car might be 17.9 miles per gallon (mpg) and 24.1 mpg for a different 
model. Note that the number of attribute choices need not be the same 
for each attribute (six exterior color attribute choices, but only two gas 
mileage choices). And that not all alternative sets may be available or 


feasible (the dealer may not have a red car that gets 24.1 mpg). 
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Table A.1 below demonstrates the architecture between attribute 
categories and aftribute choices. 










Attribute, indexed by |. For example: 
-speed (1), range (2), esthetics (3), .... maintainability (n} 





Each Attribute need not have the same number of Attribute Choices. 










The Attribute index, i, is only for notational convenience. There is no . 


ranking or value associated with i. When the letter i is used as a subscript 
for the letters r or V, the i does imply an intermediate ranking. 







Attribute, i ss 







(Most preferred level of Attribute i) ] 


(Second Most preferred level of Aitribute i) 2 | 
(Third Most preferred level of Attribute i) ~ 3 | 
(Least preferred level of Attribute I) K | 




















Attribute Choices (within Attribute i), indexed by k andranked [£ 
in order from the Attribute Choice with the most preferred level 
of Attribute i (v, in the table) to the Attribute Choice with the 
least preferred level of Attribute i (z, in the table). 










Again, there need not be the same number of Attribute 
Choices for each Attribute. K, is the number of Attribute 
Choices in Attribute I. 
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Table A.2 below demonstrates the architecture between attributes 


and multi-atiribute decision alternative sefs. 














Attribute, indexed by i. For example: 
-speed (1), range (2), esthetics (3), .... maintainability (n) 


Each Attribute need not have the same number of Attribute Choices. 
The Attribute index, i, is only for notational convenience. There is no ranking 


or value associated with i. When the letter i is used as a subscript for the 
letters ror V, the i does imply an intermediate ranking. 









Multi-Attripute Decision Alternative Sets, | Attribute, 
(Note: The ordering of the alternative sets is arbitrary. The text below 


describes a suggested possible method of ordering the sets.) 










Multi-Attribute Decision Alternative Set ,j= 1. Set 1 contains the most 
preferred Attribute Choice from each Attribute (IF the set is feasible) 
(in the table, j =V,, V>, V3 -+ Va) 


Multi-Attribute Decision Alternative Set ,j = 2. Set 2 contains the most 
preferred Attribute Choice from each Aitribute except Attribute n (IF 
the set is feasible). The set j = 2 Attribute n element is the second most 
preferred Attribute Choice from Attribute n. 


Multi-Attribute Decision Alternative Set ,j = 3. Set 3 contains the most 
preferred Attribute Choice from each Attribute except Attribute n (IF 
the set is feasible). The set j = 3 Attrinute n element is the third most 
preferred Attribute Choice from Attribute n. 
Subsequent Multi-Attribute Decision Alternative Sets ,j = 4,5, ....m -1, 
continue in a fashion similar to the Arabic numbering system (where 
Attribute n equates to the one’s column, Attribute n-1 the ten's 
column, etc.) 
Multi-Attribute Decision Alternative Set ,j =m. Setm contains the least 
preferred Attribute Choice from each Aitribute (IF the set is feasible) 
(Inthe table, j =2Z,,Z, Z, --.Z) 






























Table A.2. Attribute and Multi-Attripbute Decision Alternative Sets Definitions 










Muhi-Attribute Decision Alternative Sets contain a single Attribute Choice from 
each Attribute. The sets are indexed by j. Sets are ordered in some logical 
fashion. For example: set j= 1 contains the Attribute Choices with the most 
preferred level of their respective Attribute (IF it is a feasible set!), while the 
set, | =m, contains the Attribute Choices with the least preferred level of their 
respective Attribute (IF it is a feasible set!}. A quasi Arabic numbering system, 
where Attribute n is the one’s column, Attribute n-1 is the fen's column, etc., Is 


the recommended ordering system. 
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APPENDIX B. LAMPS DECISION ALTERNATIVES SURVEY GUIDE AND 
RESPONSE FORM 


LAMPS Decision Alternatives Survey 


Overview: This survey is being conducted to collect thesis data. The data will be used to 
Statistically analyze methods for assessing and ranking decision alternatives that have no natural 
basis for comparison. You, the "Decision Maker", will be presented with a series of decision 
problems involving LAMPS (Light Airborne Multipurpose System) helicopter systems and invited 
to give both subjective and quantitative responses. 


Scenario: All the following questions should be considered in context with the overall mission 
of USW (Undersea Warfare); redetection phase all the way through to battle damage assessment 


(BDA). 


Step 1: Subjective Rankings: You will be presented with sets of three helicopter systems in 
each of four categories: Weapons, Navigation, Communications, and Sensors. Always keeping 
in mind the given scenario (i.e., a complete USW mission), you are to rank the three 
helicopter systems in each category from best to worst. Below is a non-LAMPS example: 


Example: 
Scenario: You receive your current salary and are out shopping for a birthday gift 
for your wife or girlfriend. 
@"System Options": 
e$5 gift 
e$20 gift 


©$100 gift 
®Most people would probably rank the "system options" for the given scenario as 
follows: 


e$5 gift 
©$20 gift 
©$100 gift 





Do you understand the task? Do you have any questions? [J] Page 1 of response form. 
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Step 2(a): Quantitative Assessment: Now that you have ranked the three helicopter system 
alternatives in each category, you will be asked to give your opinion on the relative value of how 
much better the system you ranked "best" is over the "intermediate" and "worst" systems, and _ 
how much better the "intermediate" system is over the "worst" system, always keeping in mind 
that your responses should be in context with performing the given scenario of a complete 
USW mission. 









¢Simplified example: | 
Scenario: You receive your current salary and are out shopping for a birthday gift 
for your wife or girlfriend. Same "system options" as before: 







©The es» $100 bill is_5_ times better than the anenmedaey $20 Dill. 
©The wes $100 gift is_3 times better than the antemedate) G20 gift. 


©The es $100 bill is 20 times better than the wos) $5 bill. 
The es» $100 gift is _& times better than the wos $5 gift 























©The cntermediatey $Z2O bill is 4 _ times better than the wos $5 Dill _. 
©The anenmediatey $20 gift _is_2.5 _ times better than the wos $5 gift . 








Do you understand the task? Do you have any questions? ) Page 2 of response form. 
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Step 2(b): Quantitative Assessment (cont.): The following set of questions asks your opinions 


on the relative value of how much worse the "worst" system is compared to the "best" and 
"intermediate" systems, and how much worse the "intermediate" system is compared to the "best" 
system. Again, always keep in mind that your responses should be in context with 
performing the given mission scenario. | 


¢Simplified example: 
¢Scenario: You receive your current salary and are out shopping for a birthday gift 
for your wife or girlfriend. Same "hardware options" as before: 


©The wos $5 bill is _2O times worse than the es» $100 bill. 
*The wos $5 gift is_8& times worse than the ee» $100 gift. 


©The wosy_ $5 bill is 4 times worse than the cnemedae) $20 bill 
*The wos $5 gift_is_ 2.5 times worse than the cnemediae) $20 gift_. 


©The anemediatey $20 bill is 5 times worse than the es» $100 bill. 
©The cnemedie) $20 gift _is_3 times worse than the es» $100 gift. 





Do you understand the task? Do you-have any questions? [1] Page 3 of response form. 
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Step 3: Marking a Linear Scale: You will now be presented with a line labeled at one end 
with a zero (0) and the other end with a 100. Jf zero represents an absolutely worthless 
helicopter system for the given scenario, and 100 represents the absolute best helicopter system 
imaginable for the given scenario, mark the line where you think the "best" (B), "intermediate" 
(1), and "worst" (W) systems, as ranked earlier, would fall on the line. 












eSimplified example: 
eScenario: You receive your current salary and are out shopping for a birthday gift 
for your wife or girlfriend. Same "system options" as before: 


$5 gift $20 gift $100 gift 










Best possible gift 
imaginable, given 


financial condition. 
a a a a Eine URN ON Cara i 





Do you understand the task? Do you have any questions? [J Page 4 of response form. 
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Step 4(a): Let's Make a Deal!: You are outfitted with the system you have ranked as 
"intermediate". It is yours to employ in the given scenario. However, you have the option to 
trade in your current (intermediate) system for a chance at getting the "best" system. If you take 
the gamble and win, you get the "best" system. If you take the gamble.and lose, you get the 
"worst" system. If you do not take the gamble, you keep the "intermediate" system. What is the 
minimum percentage chance of winning the "best" system you are willing to accept in order 
to take the gamble? 





























¢Simplified example: | 

eScenario: You are a college football coach. There are three seconds left in the last game of the 
regular season. Your team just scored a touchdown and is currently down by one point. Your 
"system options" are the following: 


©"Worst": Do not convert on the extra point attempt, either kick or two-point attempt, and loose 
the game. 

°"Intermediate": Kick the extra point to tie the game. 

"Best": Score a two-point conversion to win the game 


°Now, if you are guaranteed able to tie the game (your team has an outstanding kicker, he never 
misses), what is the minimum percentage chance of getting the "best" result (i.c., winning the 
game) you are willing to accept in order to take the gamble and possibly end up with the "worst" result 
(i.e., loosing the game)? 


¢Your minimum acceptable percentage chance of getting the "best" result would most likely depend 
on such factors as your win-lose record, post-season bowl game opportunities, or chances of winning 
the league or national title, etc. For example: 


elf tying the game meant holding on to the league championship and a bowl game appearance, 
while loosing the game would terminate your season, the minimum percentage chance of getting the 
"best" result (i.e., winning the game) would probably have to be very high (close to 100 percent) in 
order to take the gamble. 

On the other hand, if the only way your team could get a bowl game appearance was to win 
the game outright, you would probably be willing to take a big gamble, risk the guaranteed tie, and 
accept a rather low minimum percentage chance (close to zero percent) of getting the "best" result 
(winning the game). 


©The previous examples are probably the extremes. Depending on the consequences of a win, 
tie, or lose (e.g., school pride, coaches job, etc.), your minimum acceptable percentage chance of 
risking the guaranteed "intermediate" result for a chance at getting the "best" result would likely fall 
somewhere in between. 


For the actual survey questions, you will be asked to provide a percent figure (from 0-100, 
inclusive) . Remember to keep in mind the scenario, a complete USW mission. 


Do you understand the task? Do you have any questions? J Page 5 of response form. 
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Step 4(b): Let's Make a Deal! (Again!): Just as with the previous task, you are outfitted with 
the system you have ranked as "intermediate". It is yours to employ in the given scenario. 
However, you have the option to trade in your current (intermediate) system for a chance at 
getting the "best" system. If you take the gamble and win, you get the "best" system. If you take 
the gamble and lose, you get the "worst" system. If you do not take the gamble, you keep the 
"intermediate" system. What is the maximum percentage chance of getting the "worst" system 
you are willing to accept in order to take the gamble? 





eSimplified example: 

eScenario: Same sample scenario as before... You are a college football coach. There are three 
seconds left in the last regular game of the season. Your team just scored a touchdown and is 
currently down by one point. Your "system options" are the following: 


¢"Worst": Do not convert on the extra point attempt, either kick or two-point attempt, and loose 
the game. 

e"Intermediate": Kick the extra point to tie the game. 

e"Best": Score a two-point conversion to win the game 


®Now, if you are guaranteed able to tie the game (your team has an outstanding kicker, he never 
misses), what is the maximum percentage chance of getting the "worst" result (1.¢., loosing the 
game) you are willing to accept in order risk the guaranteed tie gambling for the win? 


¢As with the previous task, your maximum acceptable percentage chance of getting the "worst" 
result would most likely depend on such factors as your win-lose record, post-season bow! game 
opportunities, or chances of winning the league or national title, etc. For example: 


elf tying the game meant holding on to the league championship and a bowl game appearance, 
while loosing the game would terminate your season, the maximum percentage chance of getting the 
"worst" result (i.e., loosing the game) would probably have to be very small (close to zero percent) in 
order to take the gamble. 

¢QOn the other hand, if the only way your team could get a bowl game appearance was to win 
the game outright, you would probably be willing to take a big gamble, risk the guaranteed tie, and 
accept a rather high maximum percentage chance (close to 100 percent) of getting the "worst" result 
(i1.e., loosing the game). | 


©The previous examples are probably the extremes. Depending on the consequences of a win, 
tie, or lose (e.g., school pride, coaches job security, etc.), your maximum acceptable percentage 
chance of risking the guaranteed "intermediate" result gambling for the "best" result would likely fall 
somewhere 1n between. 





For the actual survey questions, you will be asked to provide a percent figure (from 0-100, 
inclusive). Remember to keep in mind the scenario, a complete USW mission. 


Do you understand the task? Do you have any questions? [J] Page 6 of response form. 
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Note: This final part of the survey should only take a few moments. You will be presented with 
a short series of questions soliciting your inputs on the relative importance of the four categories 
of systems presented earlier; namely Weapons, Communications, Navigation, and Sensors. The 
underlying scenario remains unchanged; a complete USW mission from the redetection 
phase all the way through BDA. 

Of course, all four systems categories are vital to complete a successful USW mission. 
These last few questions hope to canvass your expert opinion as to the relative importance of 


each. - 
L} e @ e ® e eo e @ ® ® e e 


Step 5: Subjective Rankings: The four helicopter systems categories, Weapons, Navigation, 
Communications, and Sensors, are to be ranked from most important (or valuable) to least 
important (or valuable) with respect to the given mission scenario (i.e., a complete USW 
mission). 


Weapons 
Navigation 


Communications 
Sensors 





Do you understand the task? Do you have any questions? J] Page 7 of response form. 
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Step 6: Pairwise Comparisons: You will now be asked to give your opinions on the relative 
value (or importance) to the given mission scenario (i.e., a complete USW mission) of the 
four systems categories, compared two at a time. Your answers will be placed in a partial matrix. 
Entries in the matrix will be numerical values of how much more important (or valuable) the row 
system category is relative to the column system category. For example, if the row system 
category is 10 times more valuable (important) for the USW mission than the column system 
category, you would enter a 10. If, on the other hand, the column category was three times more 
important than the row category, you would enter 1/3. 


Do you understand the task? 


Do you have any questions? 
L) Page 7 of response form. 
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Step 7: Unlimited Budget: You are now outfitted with the systems you ranked as "worst" 
during the first part of the survey for all four of the system categories. If you were provided a 
budget that allowed you improve the systems from "worst" to "best" in all four categories, what 
proportion (percentage) of the budget. would you allocate to each category? (Keep in mind the 
given scenario, i.e., a complete USW mission.) : 


Weapons 
Navigation 
Communications 


Sensors 


cre PTV Pr 











TOTAL 100 


Do you understand the task? Do you have any questions? L] Page 7 of response form. 
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Step 8: Let's Really Make a Deal!: You are outfitted with the "best" system in one system 
category, and the "worst" systems in the remaining three system categories. This system suite is 
yours to employ in the given scenario (complete USW mission). However, you have the 
option to trade in your current (one "best" and three "worsts") suite for a chance at getting the 
"best" systems in all four system categories. If you take the gamble and win, you get the "best" 
systems in all four system categories. The downside is, if you take the gamble and lose, you get 
the "worst" systems in all four categories. If you do not take the gamble, you keep the system 
suite you have been given (one "best" and three "worsts"). What is the minimum percentage 
chance of winning the "best" systems in all four system categories you are willing to accept in 
order to take the gamble? 








Weapon Nav Comms ___ Sensors 
Given System Suite.. 
Win Gamble............ 
Lose Gamble........... 


Do you understand the task? Do you have any questions? (©) Page 8 of response form. 
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Step 9: Decision Maker Background Information: Please answer the background information 
questions listed on page 9 of the response form. This data will be analyzed for any discernible 
patterns or trends. 
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Response Form 
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Step 1: Subjective Rankings: Rank the following sets of helicopter systems from best (1) to 
worst (3). (Again, keep in mind the given scenario, i.e., a complete USW mission.) 


Weapons: 
®Mk-46 Torpedo (Mk-46), 
Conventional 500# Depth Charge (500#), 
@Mk-50 Torpedo (Mk-50). 


Navigation: 
Global Positioning System (GPS), 
eTACAN (TACAN), 
*Doppler Radar (Dop). 


¢Communications: 
eDatalink (D/L), 
©Satellite Communications (Sat Com), 
®UHF/VHF Radio (Radio). 


eSensors: 
Magnetic Anomaly Detector (MAD), 
eSurface Search Radar (Radar), 
Forward Looking Infrared (FLIR). 
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Step 2(a): Quantitative Assessment: Give your quantitative opinions on how much better the 
"best" system is over the "intermediate" and "worst" systems, and how much better the 
"intermediate" system is over the "worst". (Again, keep in mind the given scenario, Le., 2 - 
complete USW mission.) 

Lj e e e ® ® @ ® ® & © @ e 














Weapons: Best: Intermediate: Worst: 


©The ces) 1S times better than the antermediate) 





The est) _- 1S times better than the worst) 








@ THE intermediate) 1S times better than the wor 






*Navigation: Best: Intermediate: Worst: 










©The ces 1S times better than the antermediate) 








©The Gest) 1S times better than the oor) 







© The termediate) iS times better than the worst) 






eCommunications: Best: Intermediate: Worst: 





eThe gest iS times better than the antenmediate) 











©The ces) iS times better than the worst) 











©The antermediate) 1S times better than the wor 









eSensors: Best: Intermediate: Worst: 











®The est) iS times better than the antermediate) 








©The Gest iS times better than the wort 








@ The cintermediate) iS times better than the worst) 
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uantitative Assessment (cont.): Give your quantitative opinions on how much 
worse the "worst" system is than the "best" and "intermediate" systems, and how much worse the 
"intermediate" system is than the "best". (Again, keep in mind the ne scenario, 1.€., a 


complete USW mission.) 


Cj @ e 


eWeapons: Best: 


©The (Worst) 
The (Worst) _ 


©The termediate) 


*Navigation: Best: 


©The (Worst) 
©The (Worst) 


®@T he antermediate) 


Communications: 
The (Worst) 
The (Worst) 


©The (intermediate) 


eSensors: Best: 
©The (Worst) 
eThe (Worst) 


®The (intermediate) 


Intermediate: 


times worse than the ces 
times worse than the antermediate) 


times worse than the ces 


Intermediate: 


times worse than the gest 
times worse than the antenmediate) 


times worse than the ees 


Intermediate: 


times worse than the gest) 
times worse than the antermediate) 


times worse than the ces _ 


Intermediate: 
times worse than the ces 
times worse than the termediate) 


times worse than the ces 
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Step 3: Marking a Linear Scale: Jf zero represents an absolutely worthless system for the 
given scenario, and 100 represents the absolute best system imaginable for the given scenario, 
mark the line where you think the "best" (B), "intermediate" (I), and "worst" (W) systems for 


each category, as ranked earlier, would fall on the line. (Keep in mind the given scenario, 1.e., 


a complete USW mission.) 
Lj e ® ® e © ® @ ® ® @ e & 


Weapons: Worst: Intermediate: 


*Navigation: Worst: Intermediate: 


eCommunications: Worst: Intermediate: 


eSensors: Worst: Intermediate: 


Worthless] |.-annnenn-}aonna-nnnorbannnnnnnntonnnnannnpannnnccentannennnnnpnnnnnnnnatanmnmnmnntannonnnna rt mmmmnanne| 
0 10 20 30 40 50 60 70 80 90 «100 


Best system 
imaginable 


| 
t 
{ 


1 
| 
1 
1 











Step 4(a): Let's Make a Deal!: You are outfitted with the system you have ranked as 
"intermediate". It is yours to employ in the given scenario. However, you have the option to 
trade in your current (intermediate) system for a chance at getting the "best" system. If you take 
the gamble and win, you get the "best" system. If you take the gamble and lose, you get the 
"worst" system. If you do not take the gamble, you keep the "intermediate" system. What is the 
minimum percentage chance of winning the "best" system you are willing to accept in order 


to take the gamble? (Keep in mind the given scenario, i.e., a complete USW mission.) 
[) e ® @ e ® @ ® @ e @ ® @ 


®Weapons: *You currently have the dntermediate weapon system. 
©What is the minimum percentage chance of winning the wen 
weapon system you are willing to accept in order to take the gamble (and risk 
getting the wos weapon system)? 


percent. 
Navigation: ®You currently have the antermediate) navigation system. 
©What is the minimum percentage chance of winning the aa 
navigation system you are willing to accept in order to take the gamble (and risk 


getting the worst) navigation system)? 


percent. 


eCommunications: ®You currently have the dntermediat) communications system. 
eWhat is the minimum percentage chance of winning the 
Best) communications system you are willing to accept in 
order to take the gamble (and risk getting the wor» 
communications system)? 


e percent. 


eSensors: ®You currently have the cntermediste) sensor system. 
©What is the minimum percentage chance of winning the oe 
sensor system you are willing to accept in order to take the gamble (and risk 
getting the worst) sensor system)? 


percent. 








Step 4(b): Let's Make a Deal! (Again!): You are outfitted with the system you have ranked as 
"intermediate". It is yours to employ in the given scenario. However, you have the option to 
trade in your current (intermediate) system for a chance at getting the "best" system. If you take 
the gamble and win, you get the "best" system. If you take the gamble and lose, you get the 
"worst" system. If you do not take the gamble, you keep the "intermediate" system. What is the 
maximum percentage chance of getting the "worst" system you are willing to accept in order 


to take the gamble? (Keep in mind the given scenario, i.e., a complete USW mission.) 
L} e e o ® ® ® ® @ ® e e @ 


*Weapons: ®*You currently have the cntenmediste) weapon system. 
What is the maximum percentage chance of getting the jwors 
weapon system you are willing to accept in order to risk the "intermediate" system 
gambling for the ee» weapon system? 


" percent. 


eNavigation: ®You currently have the antemediate) navigation system. 
What is the maximum percentage chance of getting the wom 
navigation system you are willing to accept in order to risk the "intermediate" 
system gambling for the aey navigation system? 


e percent. 


eCommunications: ®You currently have the dntermediste communications system. 
©What is the maximum percentage chance of getting the 
Worst communications system you are willing to accept in 
order to risk the "intermediate" system gambling for the qe 
communication system? 


e percent. 


| 
eSensors: ®You currently have the cnermediste) sensor system. 
What is the maximum percentage chance of getting the oors 
sensor system you are willing to accept in order to risk the "intermediate" system 
gambling for the jes sensor system? 


percent. 











Step 5: Subjective Rankings: The four helicopter systems categories, Weapons, Navigation, 
Communications, and Sensors, are to be ranked (1) to (4) from most important (or valuable) to 
least important (or valuable) with respect to the given mission scenario (i.e., a complete USW 
mission). 


Weapons 
Navigation 


Communications 
Sensors 
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Step 6: Pairwise Comparisons: You will now be asked to give your opinions on the relative 
value (or importance) to the given mission scenario (i.e., a complete USW mission) of the 
four systems categories, compared two at a time. Your answers will be placed in a partial 
matrix. Entries in the matnx will be numerical values of how much more important (or valuable) 
the row system category is relative to the column system category. For example, if the row 
system category 1s 10 times more valuable (important) for the USW mission than the column 
system category, you would enter a 10. If; on the other hand, the column category was three 
times more important than the row category, you would enter 1/3. 
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Step 7: Unlimited Budget: You are now outfitted with the systems you ranked as "worst" 
during the first part of the survey for all four of the system categories. If you were provided a 
budget that allowed you improve the systems from "worst" to "best" in all four categories, what 
proportion (percentage) of 
the budget would you allocate to each Weapons 
category? (Keep in mind the given Navigation 
scenario, i.e., a complete USW mission.) Communications 








Step 8: Let's Really Make a Deal!: You are outfitted with the "best" system in one system 
category, and the "worst" systems in the remaining three system categories. This system suite is 
yours to employ in the given scenario (complete USW mission). However, you have the 
option to trade in your current (one "best" and three "worsts") suite for a chance at getting the 
"best" systems in all four system categories. If you take the gamble and win, you get the "best" 
systems in all four system categories. The downside is, if you take the gamble and lose, you get 
the "worst" systems in all four categories. If you do not take the gamble, you keep the system 
suite you have been given (one "best" and three "worsts"). What is the minimum percentage 
chance of winning the "best" systems in all four system categories you are willing to accept in 
order to take the gamble? 


Comms __ Sensors © Minimum percentage 


. (Best) os (Worst) (Worst) — 
Given System Suite.. chance of winning in 


Win Gamble order to take gamble: 


(Worst) (Worst) (Worst) | (Worst) 


Weapon Nav Comms _ Sensors Minimum percentage 


. : (Worst) (Best) (Worst) (Worst) oh ng . 
= a 


Win Gamble order to take gamble: 


Lose Gamble " " ; [) 


Comms __ Sensors Minimum percentage 


(Worst) (Worst) (Best) (Worst) ; ; 
Given System Suite.. wel chance of winning in 


Win Gamble order to take gamble: 


| 


Nav Comms ___ Sensors ¢ Minimum percentage 


Given System Suite.. chance of winning in 


Win Gamble order to take gamble: 


(Worst) (Worst) (Worst) 
— hat |) 








Step 9: Decision Maker Background Information: 

¢Community / years-months experience (include FRS): @Mk-I/_s @Mk- IIT. / 
*Other military aviation experience : | | 
®Year Group: 


@LAMPS flight hours: ®Total flight hours (military): 


*Months LAMPS at sea time: Have you ever worked and/or flown with: 
Depth Charges: Y/N 
©LAMPS FRS: ®Mayport GPS: Y/N 
®North Island eSat Com: Y/N 
@Other : ¢FLIR: Y/N 
LAMPS Homeport(s): ®Mayport eSWATS Grad?: Y/N 
(Excluding FRS) ®North Island 
*Barbers Point 
e Atsugi 
®Other(s): 
®Type platform(s) on which conducted LAMPS cruise: 
¢FFG-7 eDD-1052 
®*DD-963 ®Other(s): 
©CG-47 e 


e" Actual" USW time (hours) on "non friendly" submarine(s): 
¢ Any special tactics schools you have attended (e.g., SWATS).: 


eExperience with experimental equipment / R&D tests for LAMPS applications (e.g., LIDAR, 
Magic Lantern, FLIR, RDP (Radar Data Processor), Penguin, etc.) (Keep 1t UNCLAS please): 


e Any comments on survey as a whole (No holds barred!) 


What other background questions should I have asked?: 
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Figure A.D-11 


2 
> 
c-} 
4 
w 
x 
a 
“ 
S 
> 
” 
> 
w 
ad 
= 
a?) 
@ 
E 
i 
2 
iz 
c 
© 
© 
” 
hes 
& 
w 
= 
= 
< 
o 
a= 
x 
ho 
© 
= 





90 





Weapon Attribute Marking A Linear Scale Ratios, 


“ 
2 
. ad 
o 
x 
a 
© 
x) 
w” 
bw, 
© 
® 
£ 
und 
< 
oO 
£ 
x 
aww 
a 
= 
@ 
ad 
= 
2 
= 
4 
& 
2 
Gant 
© 
2 
> 
&% 
= 


= 
= 
= 
= 
S 
> 


Weapon Attribute Pairwise Comparison Ratios, 
Fois Tow Siw 


Navigation Attribute Pairwise Comparison Ratios, 
Foi, Tow, Fi 


Figure A.D-12 


Figure A.D-13 








w 
2 
% 
x 
2 
© 
fs) 
”) 
— 
G 
o 
= 
ad 
< 
D 
= 
x 
ben 
c 
= 
2 
=} 
a 
= 
< 
“ 
S 
2 
© 
2 
= 
: 
O 


Sensors Attribute Marking A Linear Scale Ratios, 


Sensors Attribute Pairwise Comparison Ratios, 
| ois Cow Siw 





| Figure A.D-15 | 











Weapons Attribute Marking A Linear Scale Caiculated 
Indifference Probability, (V, -VW/(Vp - VW) 
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Weapon Attribute Pairwise Comparison Calculated 
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